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The reading grade level of a sample of Bell System technical documents 
was assessed using the Flesch Reading Ease Index, which estimates the reading 
skill needed to cope with a document's writing style. The reading skills of two 
groups of Bell System craft and management trainees were also estimated 
from their performance on a standardized reading test. We found that most 
of these employees had sufficient reading skill to deal with the writing styles 
of the sample documents. This observed match between readers and documents 
resulted because most of our students were proficient readers, not because the 
documents' writing styles were easy. We compared the reading skills of 
students as they entered and completed training to see whether less able 
students were eliminated during training. Results did not support training 
selection as an explanation of our readers' proficiency. While results indicated 
that many Bell System employees have the reading skills necessary to cope 
with the technical documents they read, others may find that the writing 
styles of technical documents are real barriers to success on the job. We can 
identify a document that may place an undue burden on its readers by 
estimating the likelihood of a mismatch between the document's readability 
and the reading skills of its users. 
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I. INTRODUCTION 

How well technical documents communicate with their readers 
depends in part on the style in which they are written, since the way 
ideas are expressed can influence how easily they are understood. 
Readers may find a document particularly difficult to read when its 
writing style places an undue burden on their reading skill. For 
example, use of words that are unfamiliar to readers can obscure a 
document's message. Also, a document can be unclear if ideas are 
embedded in sentences whose syntax is too complex for its readers. 
Other characteristics of style are described elsewhere in this issue of 
the Journal. * 

The stylistic difficulty of written communication can be appraised 
with the help of readability formulas. 2,3 These formulas predict the 
reading difficulty that may result from a document's writing style but 
not from its content, organization, or format. Readability formulas 
estimate the reading skill needed to cope with a document's writing 
style, as manifested in the writer's choice of surface structures. A 
readability index is calculated from measures of text features that are 
thought to be indicators of stylistic difficulty. For example, many 
formulas use the average length of a text's words and sentences to 
calculate readability. These two text measures, word length and sen- 
tence length, function primarily as indicators of the lexical and syn- 
tactic difficulty of the text. In English, shorter words are likely to be 
more familiar to readers than longer words, since shorter words tend 
to occur more frequently. Shorter sentences are apt to be easier for 
readers to process than longer sentences because shorter sentences 
tend to be less complex syntactically. 

While readability indexes are often derived from relatively simple 
formulas, they can provide useful estimates of sytlistic difficulty. 4 For 
example, readability indexes tend to agree with human judgments of 
text difficulty, 5 and it is usually less time-consuming and costly to 
calculate a readability index than to collect human judgments of 
reading ease. Readability indexes have also been shown to predict 
behavioral measures of reading ease, such as reading rate 2,6 or duration 
of eye fixations, 7 in experimental settings. The readability level of 
documents has also been related to their effective use in work 
situations 8,9 and will be discussed later in this section. 

Readability formulas have their limitations. Formulas only provide 
estimates of the stylistic difficulty of texts. These formulas are derived 
from correlations between text measures related to style and measures 
of reading difficulty such as comprehension test scores. Almost all 
formulas have been developed using a multiple regression technique 
to select the linear combination of stylistic measures that best predicts 
the scaled difficulty levels of a set of texts. Most of the commonly 
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used readability formulas, such as the Flesch and Dale-Chall formulas, 
produce estimates that are accurate to within ±1 reading grade level. 10 
The Kincaid formula 11 is slightly less accurate, having a standard error 
of 1.41 as compared to the Flesch, which has a standard error of 0.81. 12 

Readability formulas are often criticized because editorial revisions 
that produce acceptable readability indices do not necessarily result 
in a more readable text. 31314 For example, shortening words or sen- 
tences will lower a text's readability index without necessarily making 
it easier to read. This should not be surprising since readability 
formulas are derived from correlations between measurable text attri- 
butes and readers' performance on comprehension tests. 3 As pointed 
out in the preceding paragraphs, easily quantifiable text features such 
as sentence length gain their predictive power by indexing underlying 
contributors to text difficulty that are less easily quantified. Some of 
the important determinants of reading difficulty, such as syntactic 
complexity and the use of the passive voice, are discussed elsewhere 
in this issue of the Journal. 1 

Just how much difficulty a particular group of readers will have 
with a document's style depends on their level of reading skill. If a 
given reader's skill matches or exceeds that required by a document, 
its writing style alone is not likely to cause trouble for that reader, 
even though the document's subject matter may be difficult. Con- 
versely, a reader with more limited skill is likely to find that the 
document's writing style alone presents a challenge. 2-4 By comparing 
a document's readability index (an estimate of stylistic difficulty) with 
an estimate of reading skill, we can predict the difficulty a reader 
might be expected to have with a document's writing style. An estimate 
of overall reading competence can be obtained from a person's per- 
formance on a standardized reading test. These tests sample a reader's 
behavior on tasks that have been found to discriminate between skilled 
and unskilled readers. 

Estimation of the match between a document's readability and the 
reading skills of its audience can be helpful in evaluating the effec- 
tiveness of technical writing that contains job-related information. 
Many companies, such as the Bell System, rely heavily on written 
communication to support job performance. When the writing style of 
a job-related document is too difficult for its users, job performance 
may suffer in a number of ways. Studies have investigated the conse- 
quences of this mismatch between document style and reading skill. 
Jobs may take longer than usual to perform because needed informa- 
tion is hard to access. 9 Errors may increase because the descriptions 
of work procedures are unclear. 8 Perplexed readers may turn to co- 
workers for help, thus reducing their co-workers' productivity. 9 

In the present study, the match between the readability of docu- 
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ments and the reading skills of readers was estimated for a large 
sample of Bell System technical documents. These documents are a 
major source of information about the operation and maintenance of 
equipment. The readability of these documents was measured by 
computer and compared with the reading skills of two groups of Bell 
System employees who use the documents. Levels of reading skill were 
estimated from the employees' scores on a reading test. 

II. METHOD 

2.1 Readability survey 

2.1.1 Documents 

Readability measures were obtained for the text of Bell System 
technical documents called Bell System Practices (BSPs). These doc- 
uments are a major source of information about the operation and 
maintenance of equipment. They are used throughout the system by 
both craft and management personnel. There are well over 250,000 
BSPs so only a sample of these documents were analyzed. Most of 
these BSPs are typeset under computer control and consequently were 
available in computer-readable form. 

The BSPs selected for analysis were those likely to be used fre- 
quently. To meet this goal, documents were chosen from three impor- 
tant areas of company activity — outside plant, customer equipment, 
and ESS* switching equipment. The BSPs dealing with outside plant 
and customer equipment were taken in a random fashion from hand- 
books provided to craft personnel for use on the job. The BSPs about 
electronic switching systems were chosen in a random fashion from 
those describing the operation of two different systems, 1 ESS and 
2 ESS switching equipment. Table I shows the number of BSPs 
analyzed from each category of company activity. Altogether, reada- 
bility indices were obtained for 140 BSPs consisting of a total of 
378,359 words. 

2. 1.2 Readability measure 

The Flesch Reading Ease Index 12 was chosen to measure readability. 
The formula for calculating this index, given in Table II, uses two 
features of a text to predict stylistic difficulty: (1) the average length 
of a text's words in syllables, and (2) the average length of its sentences 
in words. These two measures function primarily as indicators of the 
lexical and syntactic difficulty of a passage. Available evidence 2-4 
suggests that formulas using these simple word and sentence measures 
are satisfactory predictors of the stylistic difficulty of texts. The 



* Trademark of Western Electric. 
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Table I — Number of BSPs in the readability 
survey from each category of company activity 

~~ Total 

Number Number of 
Category of BSPs Words 

Outside plant 49* 101,162* 

Customer equipment 48* 83,927* 

ESS maintenance 50 213,373 

* Seven BSPs that were in both the outside-plant and 
customer-equipment handbooks were included in the totals 
for both categories. 



Table II — Formula for the Flesch Reading Ease Index and the 
algorithm for converting a vowel count to a syllable estimate 

1. Flesch Reading Ease Index (FL): 

FL = 206.84 - 84.6 WL - 1.015 SL 

Where 
WL* is the average length of a word in syllables 
SL is the average length of a sentence in words 

2. Algorithm for estimating the syllables (SYL) in a sample of words 

SYL = 0.998 VOW - 0.343 W 

Where 
VOW is total vowels (aeiouy) 
W is total words 

* WL = SYL/W. 



Kincaid formula 11 also uses these two measures and is the basis for 
the Writer's Workbench advisory on readability. 1 The Flesch formula 
was chosen for the present survey because it has been more widely 
used than the Kincaid formula and has been shown to be one of the 
most accurate predictors of readability. 10 

The Flesch Index ranges from to 100 with a higher index repre- 
senting a more readable text. A Flesch Index can also be expressed as 
a reading grade level, which is the average school grade of readers who 
would be expected to score 75 percent on a comprehension test for the 
reading material. Originally, a point on the Flesch Index scale corre- 
sponded to one-tenth of a grade, 12 but Flesch found that his formula 
underestimated the reading grade level of more difficult passages. He 
proposed a nonlinear relationship between the Flesch Index and 
reading grade level, 15 and this relationship is used in the present study 
to assign a reading grade level to an index value. 

The Flesch Index was calculated by computer, 16,17 since the BSPs 
were in computer-readable form. A program was written to compute 
average sentence length for (complete) sentences in the text of the 
BSPs. The program was able to exclude tables, headings, and titles 
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that were flagged in the text. To calculate the average length of a word 
in syllables (WL), the program estimated the number of syllables in a 
word sample from a count of the number of vowels in that sample. 
This vowel count was then converted to syllables using the formula 18 
shown in Table II. 

When syllables are estimated from a vowel count, the presence of 
words without vowels, such as abbreviations, acronyms, and numbers, 
may be a problem. Their inclusion in the computer's estimate of 
syllables can lead to an underestimation of average word length and a 
consequent underestimation of reading difficulty. This is a problem 
when analyzing technical documents. In some of the BSPs selected 
for the present survey, over 11 percent of the words did not contain 
vowels. One solution to this problem is to base the syllable estimate 
on words that conform to the syllable estimation algorithm's assump- 
tion that a word contains at least one vowel. This solution was adopted 
in the present study and has been implemented in the Writer's 
Workbench. Only nonnumeric words with at least one vowel were used 
in the estimation of syllables from a vowel count. This solution seemed 
to work. Flesch Indexes were computed using hand counts of syllables 
for 16 BSPs that had a large number of words without vowels. These 
manually produced indexes were compared with computer-produced 
indexes. When words without vowels were omitted from the computer 
estimation of syllables, 11 of the 16 texts were assigned the same 
reading grade levels by the computer-produced and manually produced 
indexes. When words without vowels were included in the syllable 
estimation, only 4 of the 16 BSPs were classified at the same reada- 
bility level by the computer-produced and manually produced indexes. 
This solution finds precedence in Flesch's suggestion 12 that counts of 
syllables be based on words whose pronunciation is unambiguous. 

2.2 Reading skills survey 
2.2.1 Reading test 

The reading skills of Bell System personnel were estimated from 
their performance on a standardized, commercially produced reading 
test, the Nelson-Denny Reading Test, Form D. 19 This test had been 
standardized by giving it to a sample of over fifteen thousand high 
school students, representing the student population of the United 
States in 1972. The test performance of students in this sample was 
used to estimate the reading skill represented by a test score. Level of 
reading skill is often expressed as a score's reading grade level. A test 
score is assigned a reading grade level that corresponds to the average 
educational level of students who made that test score in the stand- 
ardization sample. Thus, a reader's reading grade level is the school 
grade for which his or her test score is typical. It should be emphasized 
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here that the educational level of a person and the reading grade level 
of his or her test score are not identical. 

The Nelson-Denny Reading Test was chosen for three reasons: (1) 
it can be given in a reasonably short period of time, (2) it covers a 
range of reading abilities appropriate for Bell System employees, and 
(3) it was well constructed according to standards for norm-referenced 
tests. 20 

The Nelson-Denny Test is composed of a vocabulary test and a 
comprehension test. The vocabulary test contains 100 multiple-choice 
items consisting of a sentence stem and four words that can complete 
the sentence. The comprehension test consists of 36 multiple-choice 
questions that relate to five short prose passages. Readers work on the 
vocabulary test for 10 minutes and on the comprehension test for 20 
minutes. 

Performance on the test is expressed as a total score. This score is 
the sum of two subscores: (1) the number of correct vocabulary items, 
and (2) the number of correct comprehension items multiplied by two. 
The reading grade equivalents for total scores are given in the Exam- 
iner's Manual of the Nelson-Denny Test. 19 In terms of accuracy, the 
total score has a standard error of measurement* of about 5.7. 19 
Translated into reading grade level, this standard error is approxi- 
mately a 0.5 reading grade level. 

In the present study, total scores were used in all statistical com- 
putations. However, the results were also presented as reading grade 
equivalents of test scores. Reading grade levels were used because: (1) 
they are often a more meaningful way of expressing reading skill, and 
(2) the readability of the technical documents was expressed as reading 
grade levels. 

2.2.2 Subjects 

The Nelson-Denny Reading Test was given to 471 technical man- 
agement employees and 210 craft employees. Both groups were stu- 
dents at Bell System training centers. The management students were 
taking a variety of technical courses covering such diverse subjects as 
switching systems, data management, building engineering, and fi- 
nance. The craft students were taking the final course in an ESS 
switching equipment training program that involves approximately 
700 hours of training. 

These employees were chosen because they were likely to use doc- 



* The consistency of a test can be expressed in terms of the standard error of 
measurement. Approximately two-thirds of the hypothetical "true" scores on a test 
would lie within one standard error of measurement from a score obtained on the test. 
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uments such as the BSPs sampled in the readability survey. In 
addition, students at the training centers are more readily available 
for testing than are employees in work situations. 

Each student took the Nelson-Denny Test and filled out a question- 
naire requesting information about his or her age, sex, years of formal 
education, job level, years of service in the Bell System, and region of 
the country in which he or she worked. The study was conducted in 
the students' classrooms in groups ranging from 10 to 30 students. All 
students in the selected classes participated in the study although they 
were not required to do so. To ensure the anonymity of the partici- 
pants, each student used a number to identify all test material and 
the questionnaire. 

III. RESULTS 

3. 1 Readability survey 

Figure 1 shows the distributions of Flesch Indexes for BSPs in each 
of the three categories: outside plant, customer equipment, and ESS 
maintenance. The distribution for each category is displayed as a box 
plot. The ends of a box are drawn at the upper and lower quartiles of 
the distribution, while the bar across the box represents the median. 
The extremes of the distribution are plotted as horizontal lines joined 
to the box by dashed lines. 

Figure 1 suggests that considerable reading skill is required to cope 
with the writing styles of many BSPs. Thirty-three percent of all the 
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BSPs sampled had Flesch Indexes below 50. An appreciation of the 
stylistic difficulty of these BSPs can be gained by comparing their 
readability levels with those obtained by Flesch 12 for some represent- 
ative magazine articles. Flesch found that academic journals such as 
the American Scholar or the Yale Review had Flesch Indexes ranging 
from 30 to 50, while quality magazines such as the New Yorker had 
Indexes ranging from 50 to 60. Over half of the customer-equipment 
BSPs were comparable to academic journals in their level of stylistic 
difficulty. The writing style of three-quarters of the ESS switching 
equipment maintenance BSPs was at least as difficult as that of 
quality magazines. In terms of required reading skill, the writing style 
of one-third of the BSPs would be likely to cause difficulty for readers 
with skills below the 13th reading grade level. The writing style of 
over half the documents (63 percent) would be likely to cause difficulty 
for those employees reading below the 11th reading grade level. 

3.2 Reading skills survey 
3.2. 1 Reader characteristics 

The questionnaires provided information about the students who 
took the reading test. Sixty-five percent of the technical management 
students were first-level management, 27 percent were second-level 
management, and the remaining 8 percent were third- (4 percent) and 
fifth- (4 percent) level management. As Table III shows, the technical 
management students represented a broader range of age, education, 
and years of service than did the craft students. The two groups 
differed primarily in educational level. More technical management 

Table III — Age, formal education, and years 
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students (69 percent) than craft students (42 percent) had continued 
on to college after graduating from high school. There were also more 
women among the technical management students (30 percent) than 
among the craft students (9 percent). 

3.2.2 Reading scores 

The cumulative probability distributions of total scores on the 
Nelson-Denny Reading Test are shown in Fig. 2 for each group of 
students. The reading grade equivalents of test scores appear at the 
top of the figure. The median, upper, and lower quartiles of the 
distributions are also shown. The maximum total score is 172, while 
the highest reading grade level that can be assigned to a score is 15. 

Test scores indicate that, on the average, both groups of students 
were competent readers. Mean scores were 81 and 94 for craft and 
technical management students, respectively. For comparison, the 
Nelson-Denny manual 19 gives mean scores for samples of freshmen, 
sophomores, juniors, and seniors in college. These scores were 75, 85, 
93, and 98, respectively. Over half the craft and technical management 
students had test scores at or above the 14th (college sophomore) 
reading grade level. The reading grade levels of the mean scores were 
14.0 and 14.9 for craft and technical management students, respec- 
tively, and this difference between student groups was greater than 
would be expected by chance (t = 3.24, df = 671, p < 0.05). 
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3.3 Estimating the stylistic difficulty of BSPs 

An estimate of the difficulty a group of readers may have with the 
writing style of a specific document can be made by finding the 
proportion of readers whose reading grade levels on a reading test are 
below the reading grade level of the document. If a document's reading 
grade level is 12, for example, and one third of its users have reading 
skills below the 12th reading grade level, then the likelihood of a 
mismatch is 0.33 (e.g., the probability that the users' reading skills 
will be below the 12th reading grade level). 

This likelihood of a mismatch between audience and document can 
be extended to a set of documents. In this case, two measures are 
needed for each reading grade level represented by the set of docu- 
ments: 

1. The proportion of documents in the set whose reading grade 
levels are greater than a specified reading grade level (that is, the 
proportion of documents likely to be difficult for readers at that grade 
level); 

2. The proportion of readers in the group at that reading grade 
level. 

The likelihood of a mismatch is the sum of the products of these two 
estimates over all reading grade levels. This likelihood is the proba- 
bility that a member of the audience will select a document from the 
set that exceeds his or her reading grade level. 

The likelihood of a mismatch between BSPs and the technical 
management and craft students were calculated from the information 
in Table IV. This table shows the proportion of BSPs whose reading 
grade levels exceeded each level in the table. Also shown are the 
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proportion of technical management and craft students at each reading 
grade level. 

The likelihoods of a mismatch between BSPs and the student 
audiences was 0.14 for technical management students and 0.20 for 
craft students. This result suggests that technical management stu- 
dents were likely to have some difficulty with the writing style of 
about 14 percent of the BSPs in the sample, and craft students with 
about 20 percent. These estimates of difficulty suggest that the mis- 
match between students and BSPs is not great, but we do not really 
know how to interpret such results. For one thing, the Flesch Reading 
Ease Index assumes 75-percent reading comprehension. For technical 
documents, such as these BSPs, comprehension closer to 100 percent 
is probably desirable. In addition, although losses in job efficiency may 
result when employees have difficulty with a document, simplifying 
its writing style can be both costly and time-consuming. At present 
we know that it is useful to match documents to readers but we don't 
know how to calculate the benefits and costs involved in achieving 
this match. 

IV. DISCUSSION 

The results of the present study suggest that the writing style of 
many Bell System documents used on the job and in training requires 
considerable reading skill. Yet, the reading skill levels of the majority 
of the employees we tested matched or exceeded the estimate of 
required reading skill provided by the readability analysis. Our two 
samples of Bell System employees appeared to have fewer people with 
limited reading skills than might be expected from their educational 
levels. This was especially so for the craft students who had an average 
reading grade level (level 14 — college sophomore) that was two grade 
levels above their average educational level (level 12 — high school 
graduate). 

A number of selection factors may have acted to make our sample 
of Bell System employees such able readers. One of these factors may 
have been the documents that these students read during training. 
The craft students we tested had successfully completed a series of 
courses (approximately 700 hours of training) that required the use of 
highly technical documents, including BSPs. These technical docu- 
ments may have acted as selective filters, tending to eliminate students 
with limited reading skills. This explanation assumes that students 
who have reading difficulties or who are averse to reading difficult 
documents will tend to leave training. If our filter explanation is 
correct, we would expect to find proportionately more skilled readers 
among students who were about to complete training than among 
students who were starting a training sequence. We explored this filter 
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explanation by comparing the reading-test performance of craft stu- 
dents who were just beginning an ESS switching equipment training 
sequence with students who were about to complete this sequence. 

The Nelson-Denny Reading Test and a questionnaire were admin- 
istered to 160 craft employees who were enrolled in introductory ESS 
switching equipment courses given at the training centers of several 
operating companies. Their performance was compared with that of 
the 210 craft students discussed earlier (Fig. 2), who were finishing 
their last course in the ESS training sequence. 

The distributions of total scores on the reading test are shown in 
Fig. 3 for the beginning and advanced students. There was little 
difference between these two groups in their performance on the 
reading test. These results do not support the contention that training 
documents tend to filter out less able readers during ESS training, 
but rather they suggest that our sample of Bell System craft students 
was not biased by selective factors during ESS switching equipment 
training. Thus, their level of reading skill is likely to be representative 
of Bell System craft employees with similar training. 

Training documents may not have worked as filters because of the 
procedures for selecting employees who enter ESS switching equip- 
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ment training. These procedures, themselves, may lead to the choice 
of able readers. Operating companies undoubtedly choose their more 
able employees for ESS training because it is expensive and time- 
consuming. In addition, all those who start ESS training must com- 
plete a mini-course that helps to select employees who are likely to do 
well in ESS switching equipment self-instructional training. Even the 
perceived mental demands of operating and maintaining a complex 
switching system may discourage employees with limited intellectual 
skills. Although ESS trainees are not chosen explicitly for their reading 
skills, all the factors mentioned above could lead to the selection of 
competent readers as trainees. 

V. CONCLUSION 

In the present study, the reading skills of the majority of employees 
we tested were likely to be matched to the reading demands made by 
the writing styles of the majority of the technical documents they read. 
This match resulted because so many of the employees were proficient 
readers and not because the documents were easy to read. In many 
technical areas, such as ESS switching equipment, readers may have 
more reading skill than would be expected on the basis of their 
education. One implication of this finding is that demographic infor- 
mation about education may not be sufficient to predict an audience's 
level of reading skill. More direct estimates of reading skill may be 
needed, such as the Nelson-Denny Reading Test. 

Although we found a reasonable match in grade levels between 
documents and readers in the present study, this does not ensure that 
the documents are easy to read. Documents can be identified as 
difficult but not necessarily as easy to read, because the readability 
index does not include such important determinants of difficulty as 
organization, content, or format. For this reason, our findings should 
not discourage Bell System writers from writing as clearly and simply 
as possible. For many employees, the writing styles of technical doc- 
uments may be a real barrier to success on the job. Screening devices 
that combine information about the readability of documents and the 
reading skills of their audiences, such as the likelihood of a mismatch, 
can help identify documents that may be too difficult for their intended 
readers. 
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